Stream processing typically follows the pattern of continuous queries. Continuous queries are queries that execute for an indefinite amount of time on data that can change very rapidly. Such rapidly changing data are called streams, and streams oftentimes comprise events. Streams exist in many real-world scenarios, e.g. temperature readings from sensors placed in warehouses or on trucks, weather data, entrance control systems (where, for example, events are generated whenever a person enters or leaves), etc. The meanings of these and/or other terms herein may be clarified by referring to the “Event Processing Glossary,” published by the “Event-Processing Technical Society” (EP-TS), the entire contents of which is hereby incorporated herein by reference.
One typical scenario where continuous queries sometimes are implemented involves the monitoring or surveillance of systems that sometimes are quite complex in nature. A multitude of streams may exist in such an example scenario. However, it will be appreciated that not all events are of equal importance in all such scenarios. For example, in a power plant, the temperature data of the cooling liquid tends to be less important than a security breach. Certain conditions may, however, shift these priorities. For example, very high temperature data, temperature data in the context of low amounts of cooling fluid, etc., may shift priorities.
In many systems, there is a risk that important data may be processed too late to guarantee a timely reaction. Such problems may be caused, for example, by the sheer amount of data arriving in the streams, a lack of resources for handling such data, etc. Although sometimes the delay will not be too problematic, it will be appreciated that there are many scenarios where the failure to timely process large amounts of data based on shifting priorities might become quite serious. For instance, a business enterprise may not be able to timely deliver needed goods and/or services. Of course, the failure to adjust to shifting priorities in the power plant scenario, for example, may be catastrophic.
There are several currently available systems for continuous query processing. TIBCO®, for example, describes what may be termed a “query system.” Other query systems are described, for example, in U.S. Publication Nos. 2009/0292877 and 2009/0292759, the entire contents of which are hereby incorporated by reference in their entireties. TIBCO®, for example, describes one example query system indicating that:                “At runtime each declaration matches against every instance (and the rule is therefore effectively compared to every combination of instances—or RuleVariable tuples). Example: a declaration of CustomerCallEvent and ActiveCustomer for 2 CustomerCallEvents and 50 Customers will create 100 possible combinations or tuples to be considered. Adding a declaration of CustomerOrder with 100 instances increases this to 10,000 combinations or tuples.        This illustrates both the flexibility and potential cost of production rules—although of course the Rete algorithm can scale to very large tuple numbers.”        
The TIBCO® example continues by stating that:
“In addition, rules may have names, priority numbers and other metadata. Rule priorities are to aid in ‘conflict resolution’, a fancy term for what happens when 2 or more rules are valid to fire: which do we want to fire first?”
In the TIBCO® example system, events are fully processed in any case, and there are no apparent means to process important events in a prioritized fashion. Thus, the above-described and other conventional query systems unfortunately do not provide flexible mechanisms that help ensure increased (and sometimes even maximum) data processing while also caring for timely processing of events at the same time.
Thus, it will be appreciated that there is a need in the art for techniques that overcome these and/or other disadvantages. For example, it will be appreciated that there is a need in the art for techniques that provide flexible mechanisms that help ensure increased (and sometimes even maximum) data processing while also caring for timely processing of events at the same time.
One aspect of certain example embodiments relates to the attachment of priorities and/or reaction time limits to various entities of the system such as, for example, events, event types, queries, etc.
Another aspect of certain example embodiments relates to using priorities attached to system entities to tailor the system's processing behavior to match these boundary conditions while at the same time increasing (and sometimes even maximizing) the rate of events processed.
Advantageously, the system may be made to adapt its behavior to the current situation, which is changeable and may even be changing quite frequently, e.g., as in connection with a potentially rapidly changing stream. Users may in certain example embodiments specify policies to control this adaptation in certain example embodiments and, thus, in certain example instances, events (including events of special interest) may be handled appropriately, even in response to changing conditions.
It will be appreciated that in some cases, the appropriate handling of events may involve the at least temporary suspension of some routine activities while events of interest (e.g., “business critical” or “mission critical” events) are processed according to their priorities.
In certain example embodiments, a method for handling a stream of events is provided. The stream of events is received, with at least some of said events having boundary conditions attached thereto, and with the boundary conditions being maximum reaction times and/or priorities. Predefined queries are executed, via at least one processor of a computer system, on the events. For each said event having a maximum reaction time and/or priority attached thereto: whether the event can be processed within the attached boundary condition is estimated, the predefined queries are processed according to a first mode when the event can be processed within the attached boundary condition, and the predefined queries are processed according to a second mode when the event cannot be processed within the attached boundary condition. The second mode is practiced by at least temporarily suspending queries that do no consume events with attached boundary condition instead processing other queries. The second mode is ended and processing proceeds according to the first mode when it is estimated that unconsumed events having attached boundary conditions can be processed within their attached boundary conditions.
In certain example embodiments, a method for configuring a system to handle a stream of events is provided. A plurality of event types is defined. Said event types are stored in an event type registry. The system is able to execute in first and second modes. The stream of events is received, with at least some of said events having boundary conditions attached thereto, and with the boundary conditions being maximum reaction times and/or priorities. Queries are executed, via at least one processor of the system, on the events. For each said event having a maximum reaction time and/or priority attached thereto: whether the event can be processed within the attached boundary condition is estimated, the queries are processed according to the first mode when the event can be processed within the attached boundary condition, and the queries are processed according to the second mode when the event cannot be processed within the attached boundary condition. The second mode is practiced by at least temporarily suspending queries that do no consume events with attached boundary condition instead processing other queries. The second mode is ended and processing is returned to the first mode when it is estimated that unconsumed events having attached boundary conditions can be processed within their attached boundary conditions.
In certain example embodiments, corresponding systems for providing, configuring, and/or executing the above may be provided in addition or in the alternative.
In certain example embodiments, corresponding computer readable storage media tangibly storing instructions for executing such methods also may be provided.
These aspects and example embodiments may be used separately and/or applied in various combinations to achieve yet further embodiments of this technology.